Failure resistant distributed computing system

ABSTRACT

A failure resistant distributed computing system includes primary and secondary datacenters each comprising a plurality of computerized servers. A control center selects orchestrations from a predefined list and transmits the orchestrations to the datacenters. Transmitted orchestrations include less than all machine-readable actions necessary to execute the orchestrations. The datacenters execute each received orchestration by referencing a full set of actions corresponding to the received orchestration as previously stored or programmed into the computerized server and executing the referenced full set of actions. At least one of the orchestrations comprises a failover operation from the primary datacenter to the secondary datacenter. Failover shifts performance of task from a set of processing nodes of the primary datacenter to a set of processing nodes of the secondary datacenter, such tasks including managing storage accessible by one or more remote clients and running programs on behalf of remote clients.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.15/376,184, filed Dec. 12, 2016, which is a continuation of U.S.application Ser. No. 14/985,740, filed Dec. 31, 2015, which claims thebenefit of U.S. Provisional Application No. 62/098,430, filed Dec. 31,2014, entitled, “Failure Resistant Distributed Computing System”, hereinincorporated by reference.

TECHNICAL FIELD

The present disclosure is generally related to information technology,and in particular to a failure resistant distributed computing system.

BACKGROUND

Modern computing systems often include large systems including multipleservers or processors communicating over local-area or wide-areanetworks serving multiple clients. These systems can store very largeamounts of data and process many transactions in a given time period.Maintaining optimal system performance and the collection and analysisof transactional and dimensional data can be difficult or suboptimal incurrent systems.

SUMMARY

Disclosed herein are implementations of systems, methods, andapparatuses for providing a failure resistant distributed computingsystem.

According to an implementation, a failure resistant network-baseddistributed computing system with a plurality of datacenters comprisingprimary and secondary datacenters, each datacenter comprising aplurality of computerized servers, each of the computerized serverscomprising a processor, a communications port connected to a network, amemory comprising instructions executable by the processor, and amessaging queue connected via the communications port with thecomputerized servers of the datacenter, wherein the processor isconfigured to execute a processing node, and the messaging queues of theprimary and secondary datacenters are communicatively interconnected viatheir respective communication ports by one or more links, the systemfurther comprising a control center comprising one or more digital dataprocessing machines, a communications port, a memory, and a transmitterthat communicates via signals sent over its communications port coupledto the at least one messaging queue of each datacenter, wherein thecontrol center is programmed to perform machine-executable operationsstored in its memory to select orchestrations from a predefined liststored in its memory, and transmit, using the transmitter, anidentification of the selected orchestrations to a server of thecomputerized servers of the primary or secondary datacenters via arespective one of the messaging queues, and wherein each of thecomputerized servers of the primary and secondary datacenters isprogrammed to perform machine-executable operations to, responsive toreceiving identification of one of the selected orchestrations from thecontrol center via one of the messaging queues, execute the identifiedorchestration using its processor by referencing a full set of actionscorresponding to the received orchestration as previously stored orprogrammed into the computerized server and executing the referencedfull set of actions on the server processor, and at least one of themachine-executable actions is to direct at least one other computerizedserver to execute prescribed tasks on its processor, and the predefinedlist of orchestrations comprises at least one machine-executableorchestration to conduct a failover operation from the primarydatacenter to the secondary datacenter, the failover operationcomprising shifting performance of tasks from a set of processing nodesof the primary datacenter to a set of processing nodes of the secondarydatacenter, the tasks comprising managing storage accessible by one ormore clients located remotely from the datacenters, and running programsof machine-implemented operations on behalf of clients remotely locatedfrom the datacenters.

According to an implementation, a computer-implemented method isprovided for operating a failure resistant distributed computing systemcomprising primary and secondary datacenters, each datacenter comprisinga plurality of computerized servers, each of the computerized serverscomprising a processor configured to execute a processing node, and eachdatacenter comprising at least one messaging queue in communication withthe computerized servers of the datacenter, wherein the messaging queuesof the primary and secondary datacenters are communicativelyinterconnected by one or more links at respective communication portsassociated with each datacenter, the system further comprising a controlcenter, the method comprising machine-executed operations of selectingwith the control center orchestrations from a predefined list stored inthe control center, transmitting via the control center anidentification of the selected orchestrations to the computerized serverof the primary or secondary datacenters via one or more of the messagingqueues, and performing operations by each of the computerized servers ofthe primary and secondary datacenters comprising receiving, via thecommunications port of the datacenter, identification of one of theselected orchestrations from the control center via one of the messagingqueues, responding to the receiving identification by executing theidentified orchestration by referencing a full set of actionscorresponding to the received orchestration as previously stored orprogrammed into the computerized server and executing the referencedfull set of actions, wherein at least one of the machine-executableactions comprises directing at least one other computerized server toexecute prescribed tasks, and the predefined list of orchestrationscomprises at least one machine-executable orchestration to conduct afailover operation from the primary datacenter to the secondarydatacenter, the failover operation comprises shifting performance of thetasks from a set of processing nodes of the primary datacenter to a setof processing nodes of the secondary datacenter, the tasks comprisingmanaging storage accessible by one or more clients located remotely fromthe datacenters, and running programs of machine-implemented operationson behalf of clients remotely located from the datacenters.

According to an implementation, a failure resistant network-baseddistributed computing system with a plurality of datacenters isprovided, comprising primary and secondary datacenters, each datacentercomprising a plurality of computerized servers, wherein each of thecomputerized servers of the primary and secondary datacenters isprogrammed to perform machine-executable operations to, responsive toreceiving identification of a selected orchestrations from a controlcenter via a messaging queue, execute the identified orchestration usingits processor by referencing a full set of actions corresponding to thereceived orchestration as previously stored or programmed into thecomputerized server and executing the referenced full set of actions onthe server processor, and at least one of the machine-executable actionsis to direct at least one other computerized server to executeprescribed tasks on its processor, and a predefined list oforchestrations comprises at least one machine-executable orchestrationto conduct a failover operation from the primary datacenter to thesecondary datacenter, the failover operation comprising shiftingperformance of tasks from a set of processing nodes of the primarydatacenter to a set of processing nodes of the secondary datacenter, thetasks comprising managing storage accessible by one or more clientslocated remotely from the datacenters, and running programs ofmachine-implemented operations on behalf of clients remotely locatedfrom the datacenters.

According to an implementation, a non-transitory computer-readablestorage medium, is provided comprising executable instructions that,when executed by a processor, facilitate performance of operations ofthe method described above.

BRIEF DESCRIPTION OF THE DRAWINGS

The description herein makes reference to the accompanying drawingswherein like reference numerals refer to like parts throughout theseveral views, and wherein:

FIG. 1 is a block diagram of an example distributed computing system.

FIG. 2 is a block diagram of an example computerized server of thedistributed computing system of FIG. 1.

FIG. 3 is a block diagram of an example high availability processingarchitecture.

FIG. 4 is a block diagram of an example internal configuration of adigital data processing machine.

FIG. 5A is a perspective view of an example digital data storage.

FIG. 5B is a perspective view of an example logic circuit.

FIG. 6 is a flow chart of example operations performed by a controlcenter and datacenter of a distributed computing system.

FIG. 7 is a flow chart of certain example operations performed bycomputerized servers of a distributed computing system.

FIG. 8 is a block diagram showing the hardware components of an exampledatacenter.

FIG. 9 is a block diagram showing the hardware components of an exampleserver.

FIG. 10 is a map showing physical locations of datacenter hardware.

FIG. 11 is a state diagram of data management operations.

FIGS. 12A-C are parts that together comprise a pictorial view of anexample status report resulting from an example data management transferoperation.

DETAILED DESCRIPTION

The nature, objectives, and advantages of the present disclosure willbecome more apparent to those skilled in the art after considering thefollowing detailed description in connection with the accompanyingdrawings.

-   1. Hardware Components and Interconnections

One implementation of the present disclosure concerns a failureresistant distributed computing system. The hardware components andinterconnections of this digital data processing system and the relatednetwork are described as follows, whereas the functionality of thesesystems are separately discussed further below.

A. Cloud Computing Environment

Cloud computing can provide various advantages over traditionalcomputing models, including the ability to allocate shared resourcesamongst many different customers. Under traditional computing models,computing resources are typically allocated to a single customer orentity and substantial portions of those resources may remain unused orunderused.

Computing resources of cloud computing infrastructure may be allocated,for example, using a multi-tenant or a single-tenant architecture. Undera multi-tenant architecture, installations or instantiations ofapplication, database, and/or other software application servers may beshared amongst multiple customers. For example, a single web server(e.g., a unitary Apache installation), application server (e.g., unitaryJava Virtual Machine) and/or a single database server catalog (e.g., aunitary MySQL catalog) may handle requests from multiple customers. In amulti-tenant architecture, data or applications used by variouscustomers can be commingled or shared. According to an implementation ofthis architecture, the application and/or database server software candistinguish between and segregate data and other information of thevarious customers using the system. For example, database recordsbelonging to a particular customer may be identified using a customer_idfield in a database table holding records for numerous customers.

Under a single-tenant infrastructure, separate web servers, applicationservers, and/or database servers are created for each customer. In otherwords, each customer will access its dedicated web server(s), will haveits transactions processed using its dedicated application server(s),and will have its data stored in its dedicated database server(s) and orcatalog(s). In a single-tenant architecture, physical hardware serversmay be shared such that multiple installations or instantiations of web,application, and/or database servers may be installed on the samephysical server. Each installation may be allocated a certain portion ofthe physical server resources, such as RAM, storage, and CPU cycles.

In an example implementation, a customer instance is composed of fourweb server instances, four application server instances, and twodatabase server instances. As previously described, each of these serverinstances may be located on different physical servers and each of theseserver instances may share resources of the different physical serverswith a number of other server instances associated with other customerinstances. The web, application, and database servers of the customerinstance can be allocated to two different datacenters to facilitatehigh availability of the applications and data provided by the servers.There may be a primary pair of web servers and application servers in afirst datacenter and a backup pair of web servers and applicationservers in a second datacenter. There may be a primary database serverin the first datacenter and a second database server in the seconddatacenter. The primary database server can replicate data to thesecondary database server. The cloud computing infrastructure can beconfigured to direct traffic to the primary pair of web servers whichcan be configured to utilize the primary pair of application servers andprimary database server respectively. In a failure scenario, thesecondary servers may be converted to primary servers.

The application servers can include a platform application, such as onewritten in Java, for example, that provides generic platformfunctionality for accessing the database servers, integrating withexternal applications, and rendering web pages and other content to betransmitted to clients. The generic platform functionality may beconfigured with metadata stored in the database server. In other words,the operation of the platform on the application server may becustomized by certain end-users of the platform without requiring theJava code of the platform application to be changed. The database serverinstances can be configured with a database configuration and schema tofacilitate the operation of the platform. For example, the databaseserver instance can be configured with various tables for storingmetadata about applications, tables/fields, menus, forms, businessrules, scripts, and custom UI elements that are used to customize theappearance and operation of the customer instance. In someimplementations, the application servers can include web serverfunctionality and the web servers can be omitted.

In an alternative implementation, a customer instance may include onlytwo application servers and one database server. In a given cloudinfrastructure system, different implementations of customer instancesmay be used for different customer instances at the same time. Otherconfigurations and implementations of customer instances may also beused.

The proper allocation of computing resources of a physical server to aninstance of a particular software server, such as a database serverinstance, can be important to the efficient and effective functioning ofthe cloud infrastructure. If too few resources are allocated,performance of the services provided to the customer using the databaseserver may be degraded. If too many resources are allocated, computingresources may be wasted as the extra allocated resources may notmeaningfully increase the performance of the services provided to thecustomer. Repeated over allocation of computing resources may requirethat additional server hardware be purchased to satisfy the overallocation, resulting in a greater than necessary cost for providing thecloud infrastructure. In current systems, the amount of possible RAM maybe constrained per physical server and the utilization of RAM may berelatively higher than other available computing resources, such asprocessing cycles (e.g., CPU) and storage (e.g., solid state andmagnetic hard disks). Thus, it may be advantageous to more preciselyallocate the amount of RAM to each database server instance due to therelative scarcity of RAM resources.

The techniques and devices described herein relate to the allocation ofcloud computing resources, and particularly, the allocation of memory(RAM) resources to database servers installed on a particular physicalserver machine. An initial allocation of RAM to a database server can begenerated and the database server can be provisioned using the initialallocation. Periodic measurements can be taken of the database servertables and buffer sizes and ratios are calculated. Based on the ratios,a desired memory allocation can be determined, for example using apre-determined lookup table of memory allocation sizes to the calculatedratios. The desired memory allocation can be compiled in a report. Thereport can include functionality to permit a user to initiate anautomated action to re-provision the database server using the desiredmemory allocation. Alternatively, the re-provisioning of the databaseserver can be initiated automatically without user interaction.

B. Overall Architecture

FIG. 1 is a block diagram of an example distributed computing system100. The system 100 includes a primary datacenter 110.1 and a secondarydatacenter 110.2 (110) (parenthetical or reference character with nodecimal point is a reference character meaning collectively or anarbitrary instance/example). The datacenters 110 are each coupled to acontrol center 102. The control center 102 is linked to one or moreclients 150.1, 150.2 (150) via a communications network 101. Broadly,the control center 102 directs operations of the datacenters 110 onbehalf of the clients 150. Some examples of these operations includehosting storage for the clients 150 and running applications for theclients 150. In one implementation, the system 100 may constitute anexample of cloud computing, performed on behalf of the client 150. Inone example, the system 100 comprises a high availability system, whereeach data center 110 comprises a massively parallel execution engine.

The control center 102 comprises at least one digital data processingmachine. This is exemplified by a server, workstation, desktop computer,notebook computer, mainframe computer, datacenter, or other hardwareappropriate to carry out the functionality described herein. The controlcenter 102 is coupled to or includes storage 103 containing a predefinedlist of machine-readable software orchestrations. Each orchestrationnames, represents, signifies, embodies, lists, or incorporates a set ofmachine-executable actions or instructions that carry out theorchestrations. According to an implementation where the orchestrationsdo not contain the corresponding machine-executable actions, then thestorage 103 may additionally contain the actions associated with eachorchestration. The functionality of the orchestration is discussed ingreater detail below. In contrast to the illustrated example, theorchestrations may instead be provided in storage (not shown) outside ofthe control center 102 but nevertheless accessible by the control center102. The storage 103 encompasses machine-readable storage devices andmedia of all types, as well as storage by virtue of being programmedinto circuitry such as an ASIC, FPGA, DSP, and such. Numerous examplesof storage and logic circuits are explained in detail below.

The control center 102 is also coupled to or includes a configurationmanagement database (CMDB) 105. The CMDB 105 comprises a databasecontaining configuration item (CI) entries for the system's 100information technology (IT) assets such as systems, software,facilities, products, network, storage, and the like. CI types may alsoinclude business types, such as organizations, people, markets,products, vendors, and partners. These assets, as represented in theCMDB 105, may be referred to as the CIs. The CMDB 105 also describes thedependencies or other relationships among the CIs. CMDBs are widelyused, and many structural and operational details of the CMDB 105 willbe apparent to those of ordinary skill in the relevant art, having thebenefit of this disclosure.

The control center 102 is linked to the clients 150 via thetelecommunications network 101. Although illustrated as a central hubfor ease of illustration, the network 101 may be implemented by any formof communication link that supports data exchange between the controlcenter 102 and the clients 150 in satisfaction of the functions andpurposes expressed herein. In this regard, the network 101 may beconfigured as an overlay network, or a bus, mesh, tree, ring, star,peer-to-peer, overlay, or any combination or permutation of these orother known networks. The network 101 or one or more subcomponentsthereof may include the public Internet or a corporate or governmentIntranet, for example. The network 101 may include one or more localarea networks, wide area networks, Intranets, Extranets, Internetworks,Wi-Fi networks, or any other suitable technology using wires,radiofrequency, microwave, satellite, cellular, optical, or othertelecommunications.

Each of the datacenters 110 includes a plurality of computerized servers112. In one example, each datacenter 110 may be provided by one or morephysical racks of computing machines. More particularly, the datacenter110.1 includes computerized servers 112.1 a and 112.1 b through 112.1 n,and the datacenter 110.2 includes computerized servers 112.2 a and 112.2b through 112.2 n, although these numbers may be increased or decreasedin practice to suit the needs and context of the implementation. Each ofthe computerized servers comprises one or more digital processingmachines. These may be exemplified by a server, workstation computer, orother hardware appropriate to carry out the functionality describedherein.

Each datacenter 110 includes a messaging queue 116 in communication withthe computerized servers of that datacenter. In the illustrated example,each datacenter's messaging queue is run, driven, supported, hosted, orotherwise provided by one of the datacenter's computerized servers. Forinstance, in the illustrated example the computerized server 112.1 a ofthe datacenter 110.1 provides a messaging queue 116.1, and thecomputerized server 112.2 a of the datacenter 110.2 provides a messagingqueue 116.2. Despite the illustrated arrangement, and according to theneeds of the particular implementation, the messaging queues 116 may beprovided by another machine or circuit (not shown) other than thecomputerized servers. In the illustrated example, each of the messagingqueues 116 may be implemented by a general or special purpose storagedevice, nonvolatile storage, volatile storage, circuit memory, RAM, orany other device, data structure, or construct adequate to satisfy thefunctionality explained herein.

Each datacenter's messaging queue 116 is connected to the control center102 via a link 118. Further links 115 couple each messaging queue 116 toall servers of the relevant datacenter 110, enabling the messaging queue116 to provide a vehicle for distributing communications from thecontrol center 102 to the various computerized servers. An interlink 130couples the messaging queues 116.1, 116.2, which, for example aids inconducting failover operations where one datacenter assumes some levelof control over the other datacenter. The foregoing links 115 andinterlink 130 may comprise one or more wires, cables, fiber optics,wireless connections, busses, backplanes, mother boards or otherconstructs to enable communications meeting the function and purposesexpressed herein. Some, none, or all of these links may constitute anetwork, which may be separate from the network 101 or share some or allfeatures with the network 101.

To provide some further illustration of the hardware of an exampledatacenter, FIG. 8 provides a block diagram showing the hardwarecomponents of an example datacenter. The example datacenter 800 includesa storage rack 802 containing various servers 804.1-5 and one or morenetwork switches such as 814.

To provide some further illustration of the hardware of an examplecomputerized server, FIG. 9 provides a block diagram showing thehardware components of an example computerized server. The examplecomputerized server 900 includes a storage enclosure 901 containing astorage area network (SAN) unit 902, networking hardware 904, CPU 906,and RAM 912. The computerized server 900 also includes one or moredigital data storage devices which in this case are exemplified by harddisk drives 908.

In one example, the datacenters 110 may be physically located ingeographically diverse locations. In this regard, FIG. 10 is a diagramshowing physical locations of datacenter hardware. As illustrated,datacenters 1002, 1004, and 1006 are located in geographically distinctsites across the United States 1000.

C. Computerized Servers

As mentioned above, each control center includes multiple computerizedservers that conduct functions such as running applications and managingdata storage on behalf of remote clients. To illustrate these in greaterdetail, FIG. 2 provides a block diagram of an example computerizedserver of the distributed computing system of FIG. 1.

Whereas FIG. 1 illustrates the largely physical architecture of thesystem 100, including hardware of the datacenters 110, FIG. 2 depictssome processing features provided by the computerized server 112hardware. Computerized server 112 is one example of an implementation ofthe servers 112 illustrated in FIG. 1. One of these features is thevarious processing nodes 204.1-n. Each processing node 204 runs anapplication program, module, or algorithm, or conducts a databasefunction on behalf of one of the remote clients 150 and according todirections of the control center 102. The functionality of the nodes isdiscussed in greater detail below. In one implementation, each node 204comprises a virtual machine instantiation performing certainmachine-executable actions.

Alternatively, a node 204 may be an application. For example, a node 204need not have its own operating system. An agent 210 is a processingfeature of the computerized server 112, which can modify or controloperations of the computerized server 112, including managing operationsand configuration of the processing nodes. In addition to managing thenodes 204, the agent 210 may also control operations of the computerizedserver 112 outside the nodes 204. For example, the agent can configurean operating system common to multiple processing nodes or server 112.

The agent 210 includes a mailbox 212. The mailbox 212 may be implementedin the agent or via operating system build-in functionality, forexample. The term “mailbox” is used without any intended limitation orconstraint as to mail or e-mail or other message formats. Broadly, themailbox 212 provides a site for receiving commands, data, messages, orother signals from the associated messaging queue 116. The mailbox 212may, for example, be implemented as a device, buffer, storage unit,nonvolatile storage, volatile storage, circuit memory, RAM, or any otherhardware, software, or combination thereof.

The agent 210 may or may not include a plugin module 211, depending onthe particular implementation. Plugin module 211 may be configured toaccept an executable module from the control center 102 to permit theagent 210 to execute tasks other than distributed orchestrations. Forexample, according to an implementation, plugin 211 could receiveplugins for performing discovery of information on server 112 or otherservers in datacenter 110, or could receive an auto-remediation pluginthat can be configured to automatically perform tasks or orchestrationsbased on information collected by agent 210 without receiving directionfrom the control center 102. The executable modules plugged into pluginmodule 211 can be configured to send and receive messages from thecontrol center 102 and other servers and/or datacenters using mailbox212, such as described elsewhere with respect to orchestrations.

The computerized server 112 also includes storage containing a pluralityof machine-executable actions 214. The machine-executable actions from214 may be stored or programmed into the computerized server 112, andnamely, contained in storage accessible by the computerized server 112,incorporated into circuitry of the computerized server 112, incorporatedinto code executable by the server 112, or other mechanisms.

For each of the orchestrations in the control center's storage 103, thecomputerized server storage 214 contains machine-readable data orinstructions representing a full set of machine-executable actionsneeded to perform the orchestration. In contrast to storage on board theserver 112, the orchestrations may instead be provided in storage (notshown) outside of, but nevertheless accessible by, the computerizedserver 112. The storage 214 encompasses machine-readable storage devicesand media of all types. In contrast to the storage of data, the storage214 further includes “storage” by virtue of being programmed into acircuitry such as an ASIC, FPGA, DSP, and such. Various examples ofstorage and logic circuits are explained in greater detail below.

D. High Availability Processing Architecture

FIG. 3 depicts a block diagram of an example high availabilityprocessing architecture. The illustrated distributed computing system300 provides an alternate depiction of the components of FIGS. 1-2, withgreater emphasis on failure resistant features of the system. Broadly,the system 300 includes proxy-load balancers 304.1, 304.2 anddatacenters 110.1, 110.2. The proxy/load balancers 304 are coupled to acommunications network graphically depicted by the cloud 101. The cloud101 may be satisfied by the components of the network 101 as discussedabove.

The datacenter 110.1 includes a primary database 310.1, and thedatacenter 110.2 includes a secondary database 310.2. The datacenters110 operate in such a manner that the secondary database 310.2 canprovide an exact or substantially exact mirror of the primary database310.1. A line 320 is used to graphically emphasize the logical boundarybetween datacenters 110. Depending upon the intended application, aswill be apparent to those of ordinary skill in the relevant art, thedatabases 310 may range from mere digital data storage to a databasemanagement system (DBMS).

Each datacenter 110 includes two application nodes 204.1 a, 204.1 b,204.2 a, and 204.2 b (204), although a greater or lesser number may beimplemented in practice. The application nodes 204 are processingthreads, modules, virtual machine instantiations, or other computingfeatures of the datacenters 110 that run programs on behalf of remotelysited clients 150, and exchange related data with such clients 150 viathe cloud 101. In connection with running these programs, occasionsarise for the application nodes 204 to store and retrieve data, with thedatabases 310 filling this role. According to an implementation, each ofthe application nodes 204 connect to a single primary database 310.1,regardless of whether the database 310.1 is located in the samedatacenter 110.1 as the application nodes 204.1 or not. For example, aprimary database 310.1 may be read/write and a secondary database 310.2may be configured to be read-only such that it mirrors changes from theprimary database. Requests to the system 300 may be routed to theapplication nodes 204.1 in the datacenter 110.1 of the primary database310.1 first, followed by the other datacenter 110.2. In a failoversituation, the secondary database 310.2 may become read/write with theformerly primary database 310.1 switched to mirror the secondarydatabase (which becomes the primary database). In this situation, eachapplication node 204 can be reconfigured to point to the secondarydatabase 310.2 (now the primary database) as shown by the dashed lines.

As mentioned above, each datacenter 110 may have its own component304.1, 304.2 (304) that has a proxy-load balancer. Each load balancer304 may be configured to direct traffic to respective servers 112 andprocessing nodes 204 located within its data center 110. In regard toproxy services, in one example the components 304 are configured toprovide a single Internet-delivered service to remote clients 150 viathe cloud 101, where this service is actually provided by a server farmcomprising of the computerized servers 112 of the datacenters 110. Thecomponents 304 also coordinate requests from remote clients 150 to thedatacenters 110, simplifying client access by masking the internalconfiguration of the datacenters 110. The components 304 may serve thesefunctions by directing clients 150 to processing nodes as configureddirectly or via DNS.

In regard to load balancing, the components 304 can be configured todirect traffic to the secondary datacenter 110.2 in the event theprimary datacenter 110.1 experiences one of many enumerated conditionspredefined as failure. The load balancing functionality of thecomponents 304 can be provided as separate components or as a singlecomponent.

E. Data Processing Implementations

The systems illustrated above include various components that may beimplemented with data processing functionality, with some examplesincluding the components 102, 105, 110, 112, 116, 204, 210, 212, 214,304, and 310. Other components of the disclosed systems may also includesmart features, and in this respect, these components may also includedata processing features. In any of these cases, such data processingfeatures may be implemented by one or more instances of hardware,software, firmware, or a subcomponent or combination of the foregoing.The hardware of these subcomponents is described in greater detailbelow.

As mentioned above, the various data processing entities of FIGS. 1-3may be implemented in different ways.

FIG. 4 is a block diagram of an example internal configuration of acomputing device 400, such as a client 150 or server 112 devicediscussed previously, including an infrastructure control server, of acomputing system. As previously described, clients 150 or servers 112may take the form of a computing system including multiple computingunits, or in the form of a single computing unit, for example, a mobilephone, a tablet computer, a laptop computer, a notebook computer, adesktop computer, a server computer and the like.

The computing device 400 can include a number of components, asillustrated in FIG. 4. CPU (or processor) 402 can be a centralprocessing unit, such as a microprocessor, and can include single ormultiple processors, each having single or multiple processing cores.Alternatively, CPU 402 can include another type of device, or multipledevices, capable of manipulating or processing information now-existingor hereafter developed. When multiple processing devices are present,they may be interconnected in any manner, including hardwired ornetworked, including wirelessly networked. Thus, the operations of CPU402 can be distributed across multiple machines that can be coupleddirectly or across a local area or other network The CPU 402 can be ageneral purpose processor or a special purpose processor.

Random Access Memory (RAM 404) can be any suitable non-permanent storagedevice that is used as memory. RAM 404 can include executableinstructions and data for immediate access by CPU 402. RAM 404 typicallycomprises one or more DRAM modules such as DDR SDRAM. Alternatively, RAM404 can include another type of device, or multiple devices, capable ofstoring data for processing by CPU 402 now-existing or hereafterdeveloped. CPU 402 can access and manipulate data in RAM 404 via bus410. The CPU 402 may utilize a cache 430 as a form of localized fastmemory for operating on data and instructions.

Storage 404 can be in the form of read only memory (ROM), a disk drive,a solid state drive, flash memory, Phase-Change Memory (PCM), or anyform of non-volatile memory designed to maintain data for some durationof time, and preferably in the event of a power loss. Storage 404 caninclude executable instructions 404A and application files/data 404Balong with other data. The executable instructions 404A can include, forexample, an operating system and one or more application programs forloading in whole or part into RAM 404 (with RAM-based executableinstructions 404A and application files/data 404B) and to be executed byCPU 402. The executable instructions 404A may be organized intoprogrammable modules or algorithms, functional programs, codes, and codesegments designed to perform various functions described herein.

The term module, as used herein, can be implemented using hardware,software, or a combination thereof. A module may form a part of a largerentity, and may itself be broken into sub-entities. When a module isimplemented using software, this software can be implemented asalgorithmic components comprising program instructions stored in amemory, the instructions designed to be executed on a processor. Theterm “module” does not require any specific form of coding structure,and functional implementations of different modules may be independentbut also may overlap and be performed by common program instructions.For example, a first module and a second module may be implemented usinga common set of program instructions without distinct boundaries betweenthe respective and/or common instructions that implement the first andsecond modules.

The operating system can be, for example, a Microsoft Windows®, Mac OSX®, or Linux®, or operating system, or can be an operating system for asmall device, such as a smart phone or tablet device, or a large device,such as a mainframe computer. The application program can include, forexample, a web browser, web server and/or database server. Applicationfiles 404B can, for example, include user files, database catalogs andconfiguration information. In an implementation, storage 404 includesinstructions to perform the discovery techniques described herein.Storage 404 may comprise one or multiple devices and may utilize one ormore types of storage, such as solid state or magnetic.

The computing device 400 can also include one or more input/outputdevices, such as a network communication unit 406 and interface 430 thatmay have a wired communication component or a wireless communicationscomponent 490, which can be coupled to CPU 402 via bus 410. The networkcommunication unit 406 can utilized any of a variety of standardizednetwork protocols, such as Ethernet, TCP/IP, to name a few of manyprotocols, to effect communications between devices. The interface 430can comprise one or more transceiver(s) that utilize the Ethernet, powerline communication (PLC), WiFi, infrared, GPRS/GSM, CDMA, etc.

A user interface 420 can include a display, positional input device(such as a mouse, touchpad, touchscreen, or the like), keyboard, orother forms of user input and output devices. The user interface 420 canbe coupled to the processor 402 via the bus 410. A graphical userinterface (GUI) 420 is specifically a user interface that allows peopleto interact with a device in a graphical. It can be broken down into aninput portion, an output portion, and a processor that manages, process,and interacts with the input and output portions. The input portion canaccept input created by elements such as a mouse, touchpad, touchscreen,or the like. The output portion of a GUI can generate input displayableon some form of a display, such as a cathode-ray tube (CRT), liquidcrystal display (LCD), and light emitting diode (LED) display, such asan organic light emitting diode (OLED) display. The display is generallyformed of a grid of pixels, each of which can take on variousillumination and optionally color values that are grouped together andarranged to form various higher-level entities (in pixel regions) on thedisplay. These pixel regions can make up icons, windows, buttons,cursors, control elements, text, and other displayable entities. Thedisplay utilizes graphical device interface that typically comprises agraphics processor specifically designed to interact with the hardwareof the display, and may accept high-level instructions from otherprocessors to reduce demands on them. The graphical device interfacetypically has its own memory that serves as a buffer and also allowsmanipulation of stored data by the graphics processor. Operation of thedisplay thus typically involves the graphics processor accessinginstructions and data stored memory to modify pixel regions on thedisplay for the user.

Other implementations of the internal configuration or architecture ofclients and servers 400 are also possible. For example, servers may omitdisplay 420. RAM 404 or storage 404 can be distributed across multiplemachines such as network-based memory or memory in multiple machinesperforming the operations of clients or servers. Although depicted hereas a single bus, bus 410 can be composed of multiple buses, that may beconnected to each other through various bridges, controllers, and/oradapters. Computing devices 400 may contain any number of sensors anddetectors that monitor the device 400 itself or the environment aroundthe device 400, or it may contain a location identification unit 460,such as a GPS or other type of location device. The computing device 400may also contain a power source 470, such as a battery, so that the unitcan operate in a self-contained manner. These may communicate with theCPU/processor 402 via the bus 410.

F. Storage and Logic Implementations

As mentioned above, various instances of digital data storage may beused, for example, to provide storage used by the systems of FIG. 1,FIG. 2, FIG. 3, and/or FIG. 4, to embody the storage 406 or RAM 404,etc. Depending upon its application, this digital data storage may beused for various functions, such as storing data and/or storingmachine-readable instructions. These instructions may themselves aid incarrying out various processing functions, or they may serve to installa software program upon a computer, where such software program isthereafter executable to perform other functions related to thisdisclosure.

In any case, the storage media may be implemented to digitally storemachine-readable signals. One example is optical storage such as CD-ROM,WORM, DVD, digital optical tape, disk storage 500 depicted in FIG. 5A,or other optical storage. Another example is direct access storage, suchas a “hard drive”, redundant array of inexpensive disks (RAID), oranother direct access storage device (DASD). Another example isserial-access storage such as magnetic or optical tape. Still otherexamples of digital data storage include electronic memory such as ROM,EPROM, flash PROM, EEPROM, memory registers, battery backed-up RAM, etc.

An example storage medium is coupled to a processor so the processor mayread information from, and write information to, the storage medium. Inthe alternative, the storage medium may be integral to the processor. Inanother example, the processor and the storage medium may reside in anASIC or other integrated circuit.

In contrast to storage media that contain machine-executableinstructions, as described above, a different example uses logiccircuitry to implement some or all of the processing features describedherein. Depending upon the particular requirements of the application inthe areas of speed, expense, tooling costs, and the like, this logic maybe implemented by constructing an application-specific integratedcircuit (ASIC) having thousands of tiny integrated transistors. Such anASIC may be implemented with CMOS, TTL, VLSI, or another suitableconstruction. Other alternatives include a digital signal processingchip (DSP), discrete circuitry (such as resistors, capacitors, diodes,inductors, transistors, and the like), field programmable gate array(FPGA), programmable logic array (PLA), programmable logic device (PLD),and the like. FIG. 5B shows an example logic circuit 510.

-   2. Operations

Having described the hardware components and interconnections of thedisclosed digital data processing system and the related network, theoperation of these components is now discussed. The operations of anymethod, process, or algorithm described in connection with theimplementations disclosed herein may be embodied directly in hardware,firmware, software executed by hardware, circuitry, or a combination ofthese.

As discussed above, the system 100 comprises a control center 102,agents such as 210 running locally on each computerized server 112, anda communication channel to connect them. Without any intendedlimitation, the datacenters 110 may also be referred to as a serverfarm. This platform allows communication among agents 210 running onvarious servers 112 inside the server farm via a queuing mechanism. Anyagent 210 can invoke one or more operations on any other agent 210,either in a synchronous or asynchronous fashion by sending one or moremessages into the targeted agent's queue 116. Every agent has its ownqueue, and the agent listens to this queue for incoming messages.According to an implementation, when the targeted agent 210 receives amessage, it invokes the necessary operation locally using a techniqueknown as reflection. Results of this executed operation are communicatedback to the initiating/orchestrating agent via a response-messagingqueue.

Any agent 210 may act as an orchestrator and send messages to otheragents 210 across different servers 112 to perform operations inparallel. Orchestrators may invoke “fire and forget” asynchronousoperations. Orchestrators may, additionally or in the alternative, waitfor invoked operations to complete on the targeted host and collectresponses. Orchestrator agents then choose to evaluate these receivedresponses before performing the next set of operations. Agents 210 mayalso safely retry invoking the same operation on the targeted agent. Atargeted agent may choose to ignore incoming messages if the similaroperation is already in progress, or reply back with the stored resultsfrom previous execution. The agents 210 may be instructed toperiodically perform repeated operations or listen for events and sendresults via the queue to any data collection agent.

The platform assumes little control or awareness of operations executedin parallel beyond its success and failure, thus making it scalable. Theplatform provides basic functionality needed in a distributed executionenvironment such as start, stop and pause of operations and agents. Italso reports and monitors health of operations and agents.

A. Control Center & Datacenter Operations

FIG. 6 is a flow chart of example operations 600 performed by a controlcenter and datacenter of a distributed computing system. For ease ofexplanation, but without any intended limitation, the example of FIG. 6is described in the specific context of FIG. 1 and FIG. 2. In thisregard, the operations 600 are performed in the context of theillustrated failure resistant distributed computing system 100. Asmentioned above, this system includes primary 110.1 and secondary 110.2datacenters, each datacenter 110 including a plurality of computerizedservers 112. Each server 112 includes one or more digital dataprocessing machines, and provides at least one processing node 204. Eachof the datacenters 110 includes at least one messaging queue 116 incommunication with the computerized servers 112 of that datacenter 110.One or more interlinks 130 interconnect the messaging queues 116.1,116.2 of the primary 110.1 and secondary 110.2 datacenters. The controlcenter 102 includes one or more digital data processing machines coupledto the messaging queues 116.

In operation 602, the control center 102 selects one or moreorchestrations from a predefined list in storage 103. Broadly, theorchestrations are selected to carry out tasks as part of the controlcenter's 102 strategy of managing the datacenters 110. Morespecifically, the orchestrations may perform tasks including, but notlimited to:

-   -   starting and stopping processing modes;    -   changing a database mode between read-only and read-write;    -   changing connection strings;    -   switching DNS entries;    -   running post-validation;    -   running discovery;    -   supporting database isolation;    -   conducting validations, including validating an application        instance, and validating topology information;    -   transferring MySQL instances and all application instances        connected to catalogs on those MySQL instances;    -   failover of MySQL instances;    -   transfer and failover of a database server and all My SQL        instances on that server;    -   transfer and failover of a rack and all database servers in that        rack;    -   transfer and failover of an entire datacenter;    -   supporting triggering operations via a user interface in a        command line program or user interface;    -   updating a CMDB state;    -   gathering topology information;    -   updating the CMDB with a validation state; and    -   forwarding validation commands to other validation executors.

Also in operation 602, the control center 102 transmits the selectedorchestrations to at least one of the computerized servers 112. In oneexample, the data center 102 broadcasts some or all messages to allcomputerized servers rather than targeting a message to the intendedserver. In a different example, some or all messages are targeted tospecific computerized servers. In the present example, where the primarydatacenter 110 is taken to be functioning properly, the control center102 transmits the orchestrations to the servers 112 via the messagingqueue 116.

For some or all of the selected orchestrations, the transmissionincludes less than all machine-executable actions necessary to executethe selected orchestration. For example, the transmission may include apartial listing of machine-executable actions of the orchestration, ormerely the name of the orchestration or identification of a program,routine, subroutine, or other set of tasks.

In operation 604, the targeted server 112 receives the selectedorchestrations from the control center 102 via the appropriate messagingqueue 116. More particularly, the messaging queue 116 as implemented bythe server 112 receives the message containing the transmittedorchestration, and the agent 210 of the server 112 forwards the messageitself or a notification of the message to the mailbox 212 of thetargeted server 112.

At least one orchestration in the predefined list of orchestrations 103represents a machine-executable orchestration to conduct a failoveroperation of the primary datacenter 110 to the secondary datacenter110.2. This failover operation shifts performance of various tasks froma set of processing nodes 204 of the primary datacenter 110.1 to a setof processing nodes of the secondary datacenter 110.2. One example ofthese shifted tasks includes managing storage accessible by remotelylocated clients 150. Another example of these tasks is running programsof machine-implemented operations on behalf of remotely located clients150, for example using an instantiation of a virtual machine.

Despite a given server 112 receiving an orchestration from the datacenter 102, the given server 112 in some cases might not perform allactions to fully carry out the orchestration. Namely, in oneimplementation, one or more of the machine-executable actionscorresponding to an orchestration may require a server 112 to transmit acommand for execution by another server. This may be referred to asdelegation.

In one implementation, the operations 600 further include operations bywhich the computerized server supplements orchestrations received in604. Namely, a computerized server 112 may respond to receipt 604 ofinstruction to execute one of the received orchestrations bytransmitting to the control center 102 a request for an updated,corrected, or expanded list of machine-executable actions necessary toexecute the received orchestration. This may be performed regularly, ona calendar schedule, or in response to the server's detection of anabbreviated, defective, stale, or insufficient orchestration.

Further expanding on the previous example, the server's 112 request foran updated list of actions may include a first version of actionsnecessary to execute the received orchestration, for example, accordingto the server's own action cache stored in 214. In response, the controlcenter 102 compares the server-submitted version against a masterversion maintained by the control center, for example in the storage103. If the submitted version and master versions differ, the controlcenter 102 prepares a message outlining changes between the versions,and transmits this message to the submitting server 112.

In a different implementation, some or all of the transmissionoperations 602 include a differences list corresponding to a givenorchestration. Thus, operation 602 may be used to carry out theabbreviated distribution of software instructions by sending changes tothe nodes instead of complete software instructions. The differenceslist may include a change log from a list of actions previouslysynchronized between control center 102 and datacenter. In this example,the server 112 performs the given orchestration by executing an amendedset of predefined machine-executable actions. The amended set ofpredefined machine-executable actions includes the full set ofpredefined machine-executable actions necessary to execute the givenorchestration from 214, and further as amended according to thedifferences list from the control center 102.

In a further example, one or more of the orchestrations from 103 includemachine-executable actions to override one or more of the full set ofmachine-executable actions from 214.

In response to receiving an orchestration, the targeted server 112performs operation 606. Mere receipt of the orchestration may constitutea tacit instruction to perform one of the received orchestrations, orthere the control center 102 may send an overt command to this effect.In operation 606, the targeted server 112 executes the receivedorchestration by referencing a full set of actions corresponding to thereceived orchestration as previously stored or programmed into thecomputerized server (and optionally updated by changes from the controlcenter 102 as detailed above) and executing the referenced full set ofactions. For example, in an example with no changes or updates from thecontrol center 102, the agent 210 of the targeted server 112 referencesthe full set of stored or programmed actions 214, and then executes theactions.

In one implementation, the subject orchestration from the control center102 includes actions executable by one of the computerized servers 122of the secondary datacenter 110.2 to begin operating the secondarydatacenter 110.2 in substitution for the primary datacenter 110.1. Forexample, these machine-executable actions may include the followingexample sequence of tasks: (1) stopping processing nodes being executedby computerized servers of the primary and secondary datacenters, (2)for all databases provided by computerized servers of the primarydatacenter, placing the databases in read-only mode, (3) for alldatabases provided by computerized servers of the secondary datacenter,placing the databases in read-write mode, (4) re-routing connections ofthe processing nodes of computerized servers of the primary datacenterto processing nodes of computerized servers of the secondary datacenter,and (5) restarting processing nodes being executed by computerizedservers of the primary and secondary datacenters.

As a more particular example, an example sequence of failover actions bythe primary datacenter include: (1) pre-validation, including a test ifthe transfer can complete, and stopping transfer if any tests fail, (2)stopping all active primary and standby nodes, (3) switching the primarydatabase to read-only mode, (4) switching the secondary database toread-write mode, making it primary, after replication lag is zero, (5)changing connection strings in application nodes to point to newprimary, (6) switching DNS entries to point to the F5 load balancercloser to new primary, (7) starting all nodes, (8) runningpost-validation, and (9) running discovery and updating CMDBauthoritative values.

To further illustrate an implementation of data management in theenvironment of FIG. 1, FIG. 11 is a state diagram of example datamanagement operations. An operator (not shown) initiates data managementoperations at a user interface 1102 (420). Examples of these operationsinclude data transfer, failover, and validation operations. The userinterface 1102 transfers the operator instructions to the control center1104. If the primary datacenter 110.1 is unavailable, the secondarydatacenter 110.2 is used. The control center 1104 begins sub-operationsby transmitting instructions to the primary or secondary datacenter 1106(110, 112), as applicable. When the data management operation ends, thecontrol center 1104 updates the CMDB with relevant information such asthe operation start time.

FIGS. 12A-C are parts that together comprise a pictorial view of anexample status report 1200 resulting from an example data managementoperation, which in this case is a data transfer. The status report 1200indicates which sub-operations were performed as part of the transfer,and the status of each operation as completed, error/failed, notstarted, or in-progress. The status report 1200 includes a number ofboxes 1202-1208. The left column boxes 1202-1204 shown in FIG. 12Acorrespond to the primary datacenter 110.1, and right column boxes1206-1208 shown in FIG. 12B correspond to the secondary datacenter110.2. Some boxes, such as 1202 and 1206, refer to datacenter nodesserving a database function. Other boxes, such as 1203, 1204, 1207, and1208 refer to datacenter nodes serving an application function. A box1205 concerns overall status and operations of the system 100.

B. Agent Operations

FIG. 7 is a flow chart of certain example operations 700 performed bycomputerized servers of a distributed computing system. For ease ofexplanation, but without any intended limitation, the example of FIG. 7is described in the specific context of the hardware of FIG. 1 and FIG.2. In the presently illustrated implementation, each of the agent'sfunctions by separately carrying out the operations 700. For ease ofillustration, the operations 700 are explained in the context of theagent 210 of the server 112.

As mentioned above, each computerized server 112 includes a mailbox 212and each of the computerized servers 112 is programmed to executecommands received at the server's mailbox 212. Therefore, in thiscontext, operation 702 illustrates the operation of the agent 210periodically or continuously monitoring the mailbox 212 for arrival ofnew messages.

Operation 703 illustrates the receipt of a message. In one example, amessage specifies an action to be performed and any applicable criteria.“Message” may include messages, commands, data, or othermachine-readable information. Messages include orchestrations from thedata center 102 and commands from another one of the servers.

Response to arrival of the new message, the agent 210 determines thecourse of action to be taken in operation 704. In the case of anorchestration, this may include identifying the full set of actionsstored in 214, supplementing or clarifying the received orchestration,processing a change or differences log of actions, etc. In the case of amessage from another one of the servers, the message itself may forexample reference machine-executable actions stored in 214.

After operation 704, the agent 210 takes action as requested. Dependingon the message contents, a proper response may be for the agent 210 toignore the message as shown in operation 706. This may be appropriatebecause, as explained above, example, the data center 110 may broadcastsome or all messages to all servers 112 rather than targeting a messageto the intended server 112. Ignoring an incoming message may also beappropriate, for example, if the server 112 or agent 210 has alreadyinitiated the task requested by the message.

If the message pertains to the agent 210, however, and the requestedaction is not already in progress, then the agent 210 initiatesappropriate actions in operation 708. In the context of actions to beperformed by the agent 210 itself, operation 708 proceeds to operation712, where one or more of the agent's processing nodes 204 execute theaction. These actions may include running programs on behalf of remoteclients 150, managing or accessing stored data on behalf of remoteclients 150, collecting and pushing data to the control center 102,cooperating with the control center 102 to carry out management of thesystem 100, and other such tasks.

As for actions to be performed by other agents, operation 708 proceedsto operation 710, where the agent 210 directs one or more other servers'agents to execute prescribed tasks. In this instance, the delegatedagent may assume the role and responsibility of transmitting furthersynchronous or asynchronous messages to other servers 112 in connectionwith the prescribed tasks of computerized servers.

In the context of an orchestration from the control center 102, oneoutcome of operation 708 is that the agent 210 may transfer control ofexecution of future actions of an orchestration to a differentcomputerized server 112. In both of operations 710 and 712, the actionsmay be initiated by sending asynchronous “send and forget” commands, orby sending synchronous commands with a coordinated follow up.

C. Other Features

The architecture and operation of the disclosed failure resistantdistributed computing system provides a number of benefits. For example,since the control center 102 broadcasts commands throughout the system100 and the agents are interchangeable, and the control center 102assumes limited or no awareness of operations executed in parallelbeyond its success and failure, the system 100 is scalable and alsoresistant to failure of any one server 112 or agent 210. This platformprovides other capabilities such as orchestration of various operationson a cluster of servers 112 across different datacenters 110, executionof synchronous and asynchronous operations with callback mechanism onremote servers, the ability to safely retry operations, and real timedata collection across different hosts. Some other advantages includeparallel failover of “Advanced Highly Available” (AHA) applicationinstances, auditing to ensure AHA of instances and alerting on auditfailures, and the collection of service intelligence of applicationinstances across datacenters 110. A further benefit is the significantflexibility afforded by providing distributed initialization and controlof orchestrations that are predefined by a central authority andpre-coded.

-   3. Other Implementations

All or a portion of implementations of the invention described hereincan be implemented using a general purpose computer/processor with acomputer program that, when executed, carries out any of the respectivetechniques, algorithms and/or instructions described herein. Inaddition, or alternatively, for example, a special purposecomputer/processor can be utilized which can contain specializedhardware for carrying out any of the techniques, algorithms, orinstructions described herein.

The implementations of computing devices as described herein (and thealgorithms, methods, instructions, etc., stored thereon and/or executedthereby) can be realized in hardware, software, or any combinationthereof. The hardware can include, for example, computers, intellectualproperty (IP) cores, application-specific integrated circuits (ASICs),programmable logic arrays, optical processors, programmable logiccontrollers, microcode, microcontrollers, servers, microprocessors,digital signal processors or any other suitable circuit. In the claims,the term “processor” should be understood as encompassing any of theforegoing hardware, either singly or in combination.

For example, one or more computing devices can include an ASIC orprogrammable logic array such as a field-programmable gate array (FPGA)configured as a special-purpose processor to perform one or more of theoperations or operations described or claimed herein. An example FPGAcan include a collection of logic blocks and random access memory (RAM)blocks that can be individually configured and/or configurablyinterconnected in order to cause the FPGA to perform certain functions.Certain FPGA's may contain other general or special purpose blocks aswell. An example FPGA can be programmed based on a hardware definitionlanguage (HDL) design, such as VHSIC Hardware Description Language orVerilog.

The implementations herein may be described in terms of functional blockcomponents and various processing operations. Such functional blocks maybe realized by any number of hardware and/or software components thatperform the specified functions. For example, the describedimplementations may employ various integrated circuit components, e.g.,memory elements, processing elements, logic elements, look-up tables,and the like, which may carry out a variety of functions under thecontrol of one or more microprocessors or other control devices.Similarly, where the elements of the described implementations areimplemented using software programming or software elements theinvention may be implemented with any programming or scripting languagesuch as C, C++, Java, assembler, or the like, with the variousalgorithms being implemented with any combination of data structures,objects, processes, routines or other programming elements. Functionalimplementations may be implemented in algorithms that execute on one ormore processors. Furthermore, the implementations of the invention couldemploy any number of conventional techniques for electronicsconfiguration, signal processing and/or control, data processing and thelike. The words “mechanism” and “element” are used broadly and are notlimited to mechanical or physical embodiments or implementations, butcan include software routines in conjunction with processors, etc.

Implementations or portions of implementations of the above disclosurecan take the form of a computer program product accessible from, forexample, a computer-usable or computer-readable medium. Acomputer-usable or computer-readable medium can be any device that can,for example, tangibly contain, store, communicate, or transport aprogram or data structure for use by or in connection with anyprocessor. The medium can be, for example, an electronic, magnetic,optical, electromagnetic, or a semiconductor device. Other suitablemediums are also available. Such computer-usable or computer-readablemedia can be referred to as non-transitory memory or media, and mayinclude RAM or other volatile memory or storage devices that may changeover time. A memory of an apparatus described herein, unless otherwisespecified, does not have to be physically contained by the apparatus,but is one that can be accessed remotely by the apparatus, and does nothave to be contiguous with other memory that might be physicallycontained by the apparatus.

Any of the individual or combined functions described herein as beingperformed as examples of the invention may be implemented using machinereadable instructions in the form of code for operation of any or anycombination of the aforementioned computational hardware. Computationalcode may be implemented in the form of one or more modules by whichindividual or combined functions can be performed as a computationaltool, the input and output data of each module being passed to/from oneor more further module during operation of the methods and systemsdescribed herein.

Information, data, and signals may be represented using a variety ofdifferent technologies and techniques. For example, any data,instructions, commands, information, signals, bits, symbols, and chipsreferenced herein may be represented by voltages, currents,electromagnetic waves, magnetic fields or particles, optical fields orparticles, other items, or a combination of the foregoing.

The word “example” is used herein to mean serving as an example,instance, or illustration. Any implementation or design described hereinas “example” is not necessarily to be construed as preferred oradvantageous over other implementations or designs. Rather, use of theword “example” is intended to present concepts in a concrete fashion. Asused in this application, the term “or” is intended to mean an inclusive“or” rather than an exclusive “or”. That is, unless specified otherwise,or clear from context, “X includes A or B” is intended to mean any ofthe natural inclusive permutations. In other words, if X includes A; Xincludes B; or X includes both A and B, then “X includes A or B” issatisfied under any of the foregoing instances. In addition, thearticles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. Moreover, use of the term “an implementation” or “oneimplementation” throughout is not intended to mean the same embodiment,implementation, or implementation unless described as such.

The particular implementations shown and described herein areillustrative examples of the invention and are not intended to otherwiselimit the scope of the invention in any way. For the sake of brevity,conventional electronics, control systems, software development andother functional implementations of the systems (and components of theindividual operating components of the systems) may not be described indetail. Furthermore, the connecting lines, or connectors shown in thevarious figures presented are intended to represent example functionalrelationships and/or physical or logical couplings between the variouselements. Many alternative or additional functional relationships,physical connections or logical connections may be present in apractical device. Moreover, no item or component is essential to thepractice of the invention unless the element is specifically describedas “essential” or “critical”.

The use of “including,” “comprising,” or “having” and variations thereofherein is meant to encompass the items listed thereafter and equivalentsthereof as well as additional items. Unless specified or limitedotherwise, the terms “mounted,” “connected,” “supported,” and “coupled”and variations thereof are used broadly and encompass both direct andindirect mountings, connections, supports, and couplings. Further,“connected” and “coupled” are not restricted to physical or mechanicalconnections or couplings.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the invention (especially in the context of thefollowing claims) should be construed to cover both the singular and theplural. Furthermore, recitation of ranges of values herein are merelyintended to serve as a shorthand method of referring individually toeach separate value falling within the range, unless otherwise indicatedherein, and each separate value is incorporated into the specificationas if it were individually recited herein. Finally, the operations ofall methods described herein are performable in any suitable orderunless otherwise indicated herein or otherwise clearly contradicted bycontext. The use of any and all examples, or example language (e.g.,“such as”) provided herein, is intended merely to better illuminate theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed.

This specification has been set forth with various headings andsubheadings. These are included to enhance readability and ease theprocess of finding and referencing material in the specification. Theseheading and subheadings are not intended, and should not be used, toaffect the interpretation of the claims or limit claim scope in any way.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedas incorporated by reference and were set forth in its entirety herein.

The above-described implementations have been described in order toallow easy understanding of the present invention and do not limit thepresent invention. To the contrary, the invention is intended to covervarious modifications and equivalent arrangements included within thescope of the appended claims, which scope is to be accorded the broadestinterpretation so as to encompass all such modifications and equivalentstructure as is permitted under the law.

1-20. (canceled)
 21. A method for performing a failover operationbetween a primary datacenter and a standby datacenter, comprising:detecting, via a control center, that a failover condition exists in theprimary datacenter; selecting, via the control center, an orchestrationfrom a predefined list of orchestrations to facilitate a failoveroperation from the primary datacenter to the standby datacenter;transmitting, via a messaging queue, an indication of the selectedorchestration to the primary datacenter, the standby datacenter, orboth; and executing the selected orchestration to perform the failoveroperation from the primary datacenter to the standby datacenter,redirecting execution of tasks from the primary datacenter to thestandby datacenter.
 22. The method, as set forth in claim 21, whereinthe standby datacenter is configured to mirror changes from the primarydatacenter prior to redirecting the execution of tasks, and wherein theprimary datacenter is configured to mirror changes from the standbydatacenter after redirecting the execution of tasks.
 23. The method, asset forth in claim 21, wherein redirecting execution of tasks from theprimary datacenter to the standby datacenter comprises: stopping allprocessing nodes being executed by computerized servers of the primarydatacenter and of the standby datacenter.
 24. The method, as set forthin claim 21, wherein the transmitting an indication of the selectedorchestration further comprises: receiving, via at least one of theprocessing nodes, a request for an updated, corrected, expanded, or acombination thereof, of actions to execute the selected orchestration.25. The method, as set forth in claim 21, wherein redirecting executionof tasks from the primary datacenter to the standby datacentercomprises: changing one or more connection strings in application nodesto point to the standby datacenter.
 26. The method, as set forth inclaim 25, wherein redirecting execution of tasks from the primarydatacenter to the standby datacenter comprises: switching one or moreDNS entries to point to a load balancer of the standby datacenter. 27.The method, as set forth in claim 21, comprising running a discoveryprocess on the standby datacenter and updating a configurationmanagement database based on the discovery process.
 28. A non-transitorytangible computer readable medium comprising instructions that whenexecuted by a processor cause the processor to: detect, via a controlcenter, that a failover condition exists in the primary datacenter;selecting, via the control center, an orchestration from a predefinedlist of orchestrations to facilitate a failover operation from theprimary datacenter to the standby datacenter; transmitting, via amessaging queue, an indication of the selected orchestration to theprimary datacenter, the secondary datacenter, or both stop, via thecontrol center, processing on all active nodes of the primary datacenterand of the standby datacenter, in accordance with the selectedorchestration; route, via the control center, one or more connections ofprocessing nodes of the primary datacenter to processing nodes of thestandby datacenter, in accordance with the selected orchestration; andrestart, via the control center, all processing nodes of the primarydatacenter and the standby datacenter.
 29. The non-transitory tangiblecomputer readable medium, as set forth in claim 28, wherein detectingthat the failover condition exists in the primary datacenter comprises:testing to determine whether a transfer to the primary datacenter cancomplete.
 30. The non-transitory tangible computer readable medium, asset forth in claim 28, wherein the instructions that when executed bythe processor cause the processor to: run, via the control center, apost validation operation.
 31. The non-transitory tangible computerreadable medium, as set forth in claim 28, wherein routing connectionsof the processing nodes of the primary datacenter to processing nodes ofthe standby datacenter comprises: changing one or more connectionstrings in application nodes to point to the standby datacenter.
 32. Thenon-transitory tangible computer readable medium, as set forth in claim31, wherein routing connections of processing nodes of the primarydatacenter to processing nodes of the standby datacenter comprises:switching one or more DNS entries to point to a load balancer of thestandby datacenter.
 33. The non-transitory tangible computer readablemedium, as set forth in claim 28, wherein the instructions, whenexecuted by the processor, cause the processor to, after restarting allnodes of the standby datacenter, run a discovery process on the standbydatacenter and update a configuration management database based on thediscovery process.
 34. A distributed computing system comprising: aprimary datacenter having a plurality of computerized servers and aplurality of databases; a secondary datacenter having a plurality ofcomputerized servers and a plurality of databases; and a control centerhaving a data processing machine and being communicatively coupled tothe primary datacenter and the secondary datacenter, wherein the dataprocessing machine of the control center is configured to provide aninstruction to perform a failover operation to at least one of thecomputerized servers of the primary datacenter and/or secondarydatacenter to cause the failover operation from the primary datacenterto the secondary datacenter to be conducted, wherein the failoveroperation comprises: detecting, via the control center, that a failovercondition exists in the primary datacenter; selecting, via the controlcenter, an orchestration from a predefined list of orchestrations tofacilitate a failover operation from the primary datacenter to thesecondary datacenter; transmitting, via a messaging queue, an indicationof the selected orchestration to the primary datacenter, the secondarydatacenter, or both stopping, via the control center, processing on theprimary datacenter and the secondary datacenter; redirecting, via thecontrol center, communications from the primary datacenter to thesecondary datacenter; and restarting, via the control center, theprimary datacenter and the secondary datacenter.
 35. The system, as setforth in claim 34, in response to the detection that the failovercondition exists, wherein the control center is configured to providethe instruction to perform the failover operation.
 36. The system, asset forth in claim 34, wherein stopping processing on the primarydatacenter and the secondary datacenter comprises: stopping allprocessing nodes being executed by the plurality of computerized serversof the primary datacenter and by the plurality of computerized serversof the secondary datacenter.
 37. The system, as set forth in claim 34,wherein the at least one of the computerized server does not perform theinstruction to perform the failover operation, and wherein the at leastone computerized server delegates the instruction to a differentcomputerized server.
 38. The system, as set forth in claim 34, whereinredirecting communications from the primary datacenter to the secondarydatacenter comprises: changing one or more connection strings inapplication nodes to point to the secondary datacenter as a new primarydatacenter.
 39. The system, as set forth in claim 38, whereinredirecting communications from the primary datacenter to the secondarydatacenter comprises: switching one or more DNS entries to point to aload balancer closer to the new primary datacenter.
 40. The system, asset forth in claim 34, wherein the failover operation comprises, afterrestarting the secondary datacenter, running a discovery process on allprocessing nodes of the secondary datacenter and updating aconfiguration management database based on the discovery process.